DEVICE AND METHOD FOR GENERATING 
A VIRTUAL MODEL OF AN INSTALLATION 

This is a Continuation of International Application PCT/DE99/01886, 
with an international filing date of June 29, 1999, the disclosure of which is 
incorporated into this application by reference. 

FIELD OF AND BACKGROUND OF THE INVENTION 

The invention relates to a device and a method for generating a virtual 
installation model as an image of a real installation. 

This real installation is, for example, a planned or existing industrial 
facility, machines, or individual components thereof In practice, it frequently 
happens that the real installations do not conform to the original plans of the 
facility. This is due, for example, to special adaptations or retrofits made 
during facility construction. In addition, the plans for these special 
adaptations or retrofits might not have included information necessary for 
further data processing. 

OBJECTS OF THE INVENTION 

It is one object of the present invention to provide a method and a 
device to generate a virtual installation model as an image of a real installation 
in a simple manner. 



SUMMARY OF THE INVENTION 

This and other objects are achieved by a device and an associated 
method for generating a virtual installation model as an image of a real 
installation. Therein, a first memory stores picture data of the real installation; 

5 a second memory stores information data of installation components of a 

component library; and a third memory stores the virtual installation model. 
An evaluation-and-control-xmit compares the information data of the 
installation components with the picture data of the real installation. This 
comparison is performed to identify identified components in the picture data 

10 as respective ones of the installation components, to derive hypotheses 

regarding the identified components in the picture data, and to generate the 
respective identified ones of the installation components in the virtual 
installation model. 

In the associated method for generating a virtual installation model as 
15 an image of a real installation, the virtual installation model is generated from 

picture data of the real installation in that installation components of a 
component library are compared with the picture data of the real installation. 
If they coincide, each identified installation component is added to the virtual 
installation model, 

20 Two data sources form the basis for generating the virtual installation 

model. The first data source contains the picture data of the real installation, 
while the second data source contains predefined installation components that 
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were used in the construction of the facility. The evaluation-and-control-unit 
performs an image analysis, i.e., the information of the picture data and of the 
predefined installation components are combined and evaluated, possibly with 
the support of a user. As soon as an installation component is identified in the 
5 picture data, the identified installation component is added to an image of the 

virtually generated installation model. Based on the picture data of the real 
installation and with the aid of a component library of the installation 
components used, the user can thus virtually recreate the real installation. This 
provides the user with an overview of the current equipment states of the 
1 0 facility. If necessary, this overview is updated if the facility is modified. 

Since the evaluation-and-control-unit analyzes the picture data, the 
information data of the installation components of the component library, the 
current state of the generation process of the virtual installation model, and/or 
additional user information, a largely automated operation mode of the device 
15 is accomplished. 

The image analysis is advantageously performed in such a way that the 
evaluation analyzes geometric information of the picture data and/or 
geometric information of the installation components of the component 
library. 

20 To ensure a clear and comprehensive user guidance and user interface, 

it is advantageous that the device has a display unit to display three views or 
three display areas. The first view or first display area displays the picture data 
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of the real installation. The second view or second display area displays the 
information data of the installation components of the component library. 
Finally, the third view or third display area displays the virtual installation 
model. 

5 The evaluation-and-control-imit controls the process of generating the 

virtual installation model, such that an installation component selected from 
the component library is dragged into a first display area of the display unit of 
the device. Thereby, a user-controlled generation of the virtual installation 
model is achieved in a simple manner. This first display area is assigned to 

1 0 display the picture data of the real installation. 

In the following manner, an assignment of tiie respective installation 
component to the "real" installation components, which are contained in the 
picture data of the real installation, is advantageously effected. The 
evaluation-and-control-imit matches an installation component from the 

15 installation component library with the mstallation components in the picture 

of the real installation. Therein, the installation component from the 
component library is selected and dragged into the first view or the first 
display area, which shows the picture of the real installation. In particular, 
geometric properties of the installation components are evaluated. After 

20 successful identification of a "real" installation component in the picture of the 

real installation, the identified "real" installation component is assigned to tiie 
above-mentioned installation component of the component library. 
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The reliability of identifying the installation components is further 
enhanced by assigning structural information to the installation components, in 
particular geometric and functional information. This information is also 
evalviated in the evaluation-and-control-unit in order to assign the installation 
5 components of the component library to the picture data. 

After successful identification of an installation component in the 
picture data of the real installation, the evaluation-and-control-unit adds the 
identified installation component to the third view or third display area of the 
display vmit, which shows the virtual installation model. Thereby, a clear 
10 overview over the current status of the process of generating the virtual 

installation model is achieved. 

Since, in one embodiment of the present invention, the evaluation-and- 
control-unit controls an automatic function, which automatically selects and 
positions installation components and which adds identified installation 

15 components to the virtual installation model, an automatic mode of operation 

of the device is achieved. Hypotheses are generated and verified by the 
evaluation-and-control-unit in order to select installation components, to 
assign installation components to the picture data of the real installation, and 
to position the assigned installation components in the virtual installation 

20 modeL Therein, structural information are taken into account, if necessary. 



5 



The picture data are recorded by a digital camera, a digital video 
camera, digitized photographs and/or data of a CAD system to produce digital 
picture data. 

By providing different views of the real installation for recording the 
digital picture data of the real installation, a comprehensive overview of the 
entire real installation is accomplished. When an installation component is 
identified, this identified installation component is assigned to all the picture 
data of the facility. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The invention will now be described and explained in greater detail by 
means of the exemplary embodiments depicted in the figures in which: 

Fig. 1 is a block diagram showing a schematic representation of a 
device for generating an installation model; 

Fig. 2 is a screen section or display area showing a digital picture of a 
real installation; 

Fig. 3 is a screen section showing a first view or first display area for 
the real installation and showing a second view or second display area for an 
installation component "tank"; 

Fig. 4 is a screen section showing a first view or first display area for 
the real installation, a second view or second display area for the installation 



component "tank", and a third view or third display area for a virtual 
installation model; 

Fig. 5 is a screen section showing a first view for the real installation 
and a second view for an installation component "valve"; 

5 Fig. 6 is a screen section showing a first view for the real installation, 

a second view for the installation component "valve", and a third view for the 
virtual installation; 

Fig. 7 is a screen section showing a first view for the real installation, 
a second view for an installation component "pipeline", and a third view for 
1 0 the virtual installation; 

Fig. 8 is a screen section shovraig a view for an installation 
component "tank" and shoving a further view depicting structural data 
assigned to the installation component "tank"; 

Fig. 9 depicts an information, operatuig and monitoring system based 
15 on the virtual installation model; and 

Fig. 10 is an embodiment of a data model for the structure of a 
component library and of the virtual installation model. 
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DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

Fig. 1 shows a block diagram of a device for generating a virtual 
installation model. Reference number 1 identifies a real installation. Image 
recording system 3 records images of the real installation 1, which are stored 

5 in a memory 20 of a device 22 for generating a virtual installation model 2. 

Picture data 4, hereinafter also referred to as digital picture data, is supplied to 
an evaluation-and-control-unit 5. In addition to the picture data 4, the 
evaluation-and-control-unit 5 processes component data 13 of a component 
library 6, which is stored in a second memory 21 of the device 22. The second 

10 memory 21 of the component library 6 includes a partial memory area 24 for 

storing structural information 23 of the installation components 6, An arrow 
14 symbolizes that the evaluation-and-control-unit 5 is capable of processing 
user data 14 of a user 7. Output data 27 at the output of the evaluation-and- 
control-unit 5 serves as input data for the virtual installation model 2. A 

15 screen or display unit 8 displays the real installation 1, the installation 

components 6, and the generated virtual installation model 2. Therein, the real 
installation 1 is represented by the picture data 4. 

The central element of the device 22 for generating the virtual image 2 
of the real installation 1 is the evaluation-and-control-unit 5. The evaluation- 
20 and-control-unit 5 performs an image analysis, wherein geometric information 

contained in the digital picture data 4 is identified and matched with geometric 
information contained in the component information 13. 
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The image analysis of the evaluation-and-control-tmit 5 determines the 
position and orientation of the individual installation components 13. If 
necessary, this process is controlled by the user by means of the user data 14. 
In each phase of generating the virtual installation 2, the user 7 is informed 
5 through the screen 8 of the current status of the virtually generated installation 

2. If necessary, the user intervenes and supports the generation process, as 
will be explained below in the context of Figs. 2 to 8. 

Fig. 2 shows a screen section that is displayed on the screen 8 of the 
device depicted in Fig. 1. A first display area 9 shows the digital picture 4 of 
10 the real installation, which is based on digital picture data. The digital picture 

4 reaches the screen 8 via the evaluation-and-control-unit 5 (see Fig. 1). The 
screen 8 is provided with icon bars 12a, 12b, 12c, which serve as an 
interaction and operating interface for the user. The icon bars each include 
control elements to select, pick, and drag picture data and/or components, for 
15 example. The right display area m an upper screen area 10 includes a window 

13 to display individual installation components. These installation 
components are selected by means of the icon bar 12a. A lower right display 
area 1 1 includes a third window 15 to display the vutual installation 2, i.e., the 
installation components that have akeady been assigned to the "real" 
20 installation via the picture data shown in window 4. Furthermore, through 

control bars 12c and 12d, the device according to the present invention is 
capable of a "camera control", i.e., a movement of the components shown in 
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the display areas 13 and 15 in 3 -dimensional space. Instead of, or in addition 
to, the icon bar 12a for selecting the installation components, a further separate 
view showing graphic components or components, which are depicted as 
objects, may be provided on the screen 8. 

5 The screen section shown in Fig. 2 is used, for example, after certain 

digital picture data 4 have been called up, and forms the startmg point of 
generating the virtual installation model. In less complex facilities, the user 
starts an automatic operation based on the "real" installation shown in the left 
display area. In the automatic operation, the mdividual components of the 
10 component library are successively called up and the evaluation-and-control- 

unit 5 attempts to assign them to the digital picture data 4. The evaluation- 
and-control-unit 5 uses a predefined search key to evaluate the information 
data assigned to the respective component, which is to be placed in the digital 
picture data 4 of the real installation. A first evaluation relates to the 
15 geometric data associated vAth this component and a second evaluation relates 

to the structural data assigned to the component. Furthermore, the generation 
of the virtual installation model is based on previously positioned components 
and the still existing gaps in the vntual installation model. For example, the 
search pattern may be limited in that, in the area of a previously placed 
20 component "tank", which includes the information data "connecting valve 

position XXX," only components with the property "valve" are checked. As a 
rule, in more complex facility structures, the operation of assigning the 
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individual components of the component library to the digital picture data 4 is 
at least partially manual, as will be explained in the context of Fig. 3 to 7. 

Fig. 3 shows a screen section depicting a first view for the real 
installation 4 and a second view for an installation component of a tank 16a. 

5 The virtual installation component 16a is the virtual image of the real tank 16b 

shown in the digital picture data 4. In the embodiment shown in Fig. 3, the 
display in display area 10 was produced by the user manipulating the menu 
bar 12a. An arrow 17a, which is indicated by a dashed line, symbolizes that 
the user drags the virtual installation component tank 16a into the left display 

10 area of the digital picture data 4 and positions it in the area of the real tank 

16b. 

The user thus selects the installation component in the component view 
10 and drags it to the view 4 (drag-and-drop procedure). 

Fig. 4 shows the next step after the virtual installation component 16a 

15 has been dropped m the left display area of the real installation 4. Part 17b 

symbolizes that the virtual installation component 16a was dragged into the 
left display area and dropped in the area of the real tank 16b. In the vicinity of 
this item, the image analysis of the evaluation-and-control-unit 5 attempts to 
match the geometric properties of the installation component 16a with the 

20 geometric properties of the screen section. For example, edges or 

combinations of edges are evaluated. After successfiil evaluation, the 
installation component 16a is, with respect to position and orientation, 
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assigned to the digital picture data 4 and labeled accordingly. At the same 
time, in the lower right display area 11, a so-called "instantiation" of the 
virtual installation component 16a is effected in the virtual installation model. 
Thus, the virtual installation object 16a appears in the installation view 
5 displayed in display area 1 1 . 

Fig. 5 shows a further example of placing an installation component in 
the area of the digital picture data 4. Here, a valve 18a, which is activated in 
the installation component library by means of the menu bar 12a, is shown in 
the upper right display area 10. By means of the menu bar 12b, the valve 18a 
10 is dragged along a line, which is identified by a dashed arrow 26a, in the 

direction of a real valve 18b. There, the valve 18a is dropped. 

Fig. 6 shows the virtual valve 18a as dropped in the left display area 9. 
In addition, the previously identified and placed virtual tank 16a is displayed. 
The thus created virtual installation view, which comprises the virtual 
15 installation tank 16a and the virtual valve 18a, is displayed in the lower right 

display area 11. 

If an installation component cannot be positioned automatically, the 
positioning and orienting of this installation component is done by the user. 
Therein, the geometric properties analyzed in the picture 4 are matched with 
20 the geometric properties of the installation component. Thereby, the position 

and orientation of the installation component is defined. If the geometric 
information available in one view of the digital picture data 4 is not sufficient 
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for an assignment, then an assignment of the respective installation component 
is attempted through other display pictures in the form of the digital picture 
data 4. 

Fig. 7, as a further example, illustrates the assignment of a pipeline 19a 
5 to the real installation, which is shown in the left view of the digital picture 

data 4. The lower right display area 11 shows which installation components 
have already been identified in the virtual view of the installation. 

Fig. 8 illustrates, by means of the example of the virtual installation 
component 16a, which is displayed in the upper right window 10, the 
1 0 assignment of additional structural data 23 shown in the left display area. This 
structural data includes, for example, information on the size and the 
connection possibilities of the tank 16a. The structural data 23 are evaluated 
when the tank is assigned to tiie digital picture data. Thus, the structural data 
23 support orientation and positioning of the respective installation 
15 component. The structural data is used, for example, to generate hypotheses 

as to the nature of an additional component and/or the location of an additional 
component. 

The individxxal installation components of the installation model 
include a reference to the picture data used for the respective generation phase 
20 of the virtual installation model, so that the virtual installation model fimctions 

effectively. 
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The installation components, in addition to knowing the reference to 
the respective picture, also know their respective position within the picture. 
The pictures, i.e., the digital picture data, themselves contain references to the 
components contained in installation model 11, which, in tum, reference the 
picture. 

Fig. 9 shows an information, operating and/or monitoring system 31 
based on the virtual installation model 2, which is generated by the device 22 
depicted in Fig. 1. The information, operating and monitoring system 31, 
hereinafter also referred to as O&M system, is coupled with the virtual 
installation model 2 via a converter 30. The O&M system is furthermore 
coupled with the real installation 1 via a bi-directional connecting circuit. The 
parts of the device 22 for generating the vhtual installation model correspond 
to those shown in the embodiment of Fig. 1. Therefore, reference is made to 
Fig. 1 with respect to the description of the device 22 and its reference 
ntimbers. 

The information 13 and 23, which is included in the virtual installation 
model 2 or in the components of the component library 6, is used for various 
downstream-connected systems. By way of example, the use for operating 
and monitoring systems (e.g., WinCC of Siemens) is shown herein. The 
converter 30 extracts and converts the information relevant to the operating 
and monitoring system 31 from tiie virtual installation model 2. Separate 
design of the operating and monitoring system is either unnecessary or 
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drastically reduced. The operating and monitoring system 3 1 is connected to 
the real installation 1 and is capable of displaying the current state of the 
process of generating the virtual installation model 2, e.g., with the aid of 3- 
dimensional visualization based on the virtual installation model Defined 
5 control elements of the components are capable of intervening the process. 

Further systems, which could be based on the virtual installation model 
include, for example, a control system, a simulation system, a diagnostic 
system, and an information system. 

It is a fundamental advantage of coupling the virtual installation model 
10 with the real installation 1 that the virtual installation 2 does not only serve for 

static visualization and documentation of the facility L Beyond that, the 
virtual mstallation 2 assumes many additional real functions with respect to 
information, operation and monitoring of the real installation. For example, 
with the aid of the virtual installation model 2, which is an exact image of the 
15 real installation 1 with respect to defined functions, hazardous areas, very 

remote areas, areas difficult to access, etc., are easily monitored v^th exact 
visualization. Based on fictitious simulation data, the virtual installation 
model is also used for shnulations, e.g., for training purposes. 

Fig. 10 shows an exemplary embodiment of a data model for the 
20 structure of the component library 6 and the virtual installation model 2, 

together with their interconnections. This structure is used, for example, in the 
device 22 depicted in Figs. 1 and 9. To the extent possible, the reference 
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numbers introduced in Figs. 1 and 9 are used for the description of Fig, 10. 
Picture data 201 (hereinafter also referred to as sources), e.g., digital pictures 
or CAD drawings, and image information 202, 203, 204, 205 associated 
therewith is stored in an image memory 20 (= first memory 20 in Figs. 1 and 
5 9). The evaluation-and-control-unit 5 converts the information contained in 

the source 201 of the image memory 20 into a prepared source 51. For a 
source assignment 52, a geometric assignment 54 is used to describe which 
geometric elements 53 of a component 61 could be matched with geometric 
elements of the prepared source. The second memory 6 of the component 
10 library includes prefabricated or predetermined components, e.g., tank, valves, 

piping, etc. (cf. description of Figs. 1 to 9). 

The third memory 2 of the virtual installation model includes the 
components 61 of the virtual installation, the information on the prepared 
sources 51, as well as the assignment information 52 between the components 
15 61 and the prepared images 20 1 . 

The data model depicted in Fig. 10 will now be described with the aid 
of UML notation (Unified Modeling Language). The notation has the 
following semantics. A so-called "class" describes an information unit, e.g., 
an information unit component. A class has one or more attributes. The 
20 attributes define the concrete properties or the state of a class or mstance 

(attribute values). For example, a class "structural information" 62 has the 
attribute "^component type." Classes create associations (relations) with other 
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classes or with themselves. A relation describes what assignments exist 
between classes in terms of a so-called "role" (e.g., analyzed components) and 
the cardinality (0..n -> no, one or several assignments). 

A rhombus is used to characterize relations that create an "include" 
role to another class, e.g., a component includes structural information 61, 
physical behavior 63, and control behavior 64. A further special relation is 
"inheritance", which is identified by a small triangle at the end of a so-called 
"superclass". "Inheritance" indicates that properties of a subclass have been 
derived from the superclass, so that the subclass inherits the properties of the 
superclass. For example, the subclasses "point" 55, "line" 56, and "curve" 57 
inherit the properties of the superclass "geometric element" 53, These 
properties include not only the attributes but also the relations and the methods 
of a class. These methods are not further described herein. The data structure 
shown in Fig. 10 is capable of using two different sources 201, 

Both source types (subclasses) describe a view onto a facility to be 
imaged. Image 202 is a source type that represents a digital image, which is 
composed of a plurality of pixels 204. CAD drawing 203 is a source type that 
represents a CAD drawing 203, which is composed of CAD elements 205 
(lines, polygons, arcs, etc.). 

The information contained in the source 201 is converted into the 
prepared source 51 by the evaluation-and-control-unit 5. Either the pixels 204 
of the image 202 or the CAD elements 205 of the CAD drawing 203 are 
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converted into the geometric elements 53 (e.g., point 55, line 56, curve 57, ... 
58). Based on the geometric elements 53, the evaluation-and-control-unit 5 
attempts to assign the selected components 61 of the prepared source 51. 

The evaluation-and-control-unit 5 tries - automatically or in interaction 
5 with the user - to identify components in the prepared sources 5 1 (the images 

202 or the CAD drawings 203) and to add identified components to the virtual 
installation model 2. This assignment is based on the geometric elements 53 
which were analyzed in the prepared source 51, or on the components 61 via 
which geometric properties 68 are assigned. 

10 If the component 61 of the prepared source 51 could be assigned, then 

this information is stored in the source assignment 52. The source assignment 
52 describes which analyzed components 61 can be assigned to which sources 
20 L One component 61 can be assigned to different sources 201 via several 
source assignments 52. 

15 For the source assignment 52, several geometric assignments 54 

describe which geometric elements 53 of a component could be matched with 
geometric elements 53 of the prepared source 51. 

The virtual installation model 2 includes those components that could 
already be analyzed. The information of the virtual installation model 2, or 
20 the components 61 contained therein, are used by different downstream- 

connected systems, e.g., an operating and monitoring system. 
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Thus, in summary, the invention relates to a method and a device for 
generating a virtual installation model as an image of a real installation. 
Digital picture data representing images of a real installation on the one hand 
and installation components of a component library on the other hand serve as 
a database. By means of an image analysis, the data of the installation 
components as well as the digital picture data of the real installation are 
evaluated. Based on this evaluation, the identified installation components are 
assigned to the virtually generated installation model. The virtual image of the 
real installation thus created documents the actual structure of the facility and 
simplifies a failure analysis in the event of a malfunction of the real 
installation. In addition to storing geometric data, the system also stores 
functional data etc. regarding the installation components. 

The above description of the preferred embodiments has been given by 
way of example. From the disclosure given, those skilled in the art will not 
only understand the present invention and its attendant advantages, but vAll 
also find apparent various changes and modifications to the structures and 
methods disclosed. It is sought, therefore, to cover all such changes and 
modifications as fall within the spirit and scope of the invention, as defined by 
the appended claims, and equivalents thereof. 
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